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1.0  Purpose  and  Background 

1.1  Report  Purpose 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  to  determine  the  utility  of  distributed  testing.  We  have  determined  that  there  is  utility, 
and  the  purpose  of  this  report  is  to  articulate  our  findings.  This  assessment  was  based  on  the 
results  and  lessons  learned  from  JADS  testing  along  with  results  from  other  related  efforts.  This 
report  is  targeted  at  the  program  manager  and  test  manager  level  of  responsibility.  Detailed 
information  to  support  the  conclusions  draw  in  this  report  is  available  in  the  reports  listed  in 
Section  6.  For  more  information  see  the  JADS  Final  Report. 

1.2  Background 

Distributed  testing  utilizes  advanced  distributed  simulation  (ADS)  to  link  together  live,  virtual 
and/or  constmctive  simulations.  Live  simulations  are  real  players  in  a  real  environment.  Open  air 
ranges  are  considered  live  simulations  because  according  to  the  Defense  Science  Board  “anything 
short  of  war  is  a  simulation.”  Virtual  simulations  are  real  players  (or  parts  of  a  real  player)  in  a 
virtual  environment.  Human-in-the-loop  and/or  hardware-in-the-loop  laboratory  facilities  are 
examples  of  virtual  simulations.  Constructive  models  have  virtual  players  in  a  virtual 
environment.  Modular  semi-automated  forces  (MODSAF),  Janus  and  Suppressor  are  examples 
of  constructive  models.  ADS  is  significantly  different  than  what  has  historically  been  referred  to 
as  modeling  and  simulation  (M&S)  in  two  important  ways.  The  common  paradigm  for  M&S  is 
constmctive  models  only  and  does  not  include  live  or  virtual  models.  The  second  significant 
difference  concerns  linking  together  simulations.  The  conventional  view  of  M&S  is  for  stand¬ 
alone  simulations.  These  differences  between  ADS  and  conventional  M&S  are  particularly 
relevant  when  using  ADS  to  support  test  and  evaluation  (T&E).  The  conventional  application  of 
M&S  to  T&E  is  to  use  a  stand-alone  constmctive  model  to  predict  system  under  test  (SUT) 
performance.  ADS  links  together  a  real  (or  partially  real)  system  under  test  to  a  synthetic  test 
environment  created  by  linking  live,  virtual,  or  constmctive  players  and  measures  system  under 
test  performance.  The  SUT  and  the  test  assets  in  the  synthetic  test  environment  can  be 
geographically  separated  by  small  or  large  distances,  thus  creating  a  distributed  test. 

JADS  was  chartered  by  the  Deputy  Director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation),1  Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  in  October 
1994  to  investigate  the  utility  of  ADS  technologies  for  support  of  T&E.  The  program  is  Air 
Force  led  with  Army  and  Navy  participation  and  is  scheduled  for  completion  in  March  2000. 

JADS  directly  investigated  ADS  applications  in  three  slices  of  the  T&E  spectrum:  the  System 
Integration  Test  (SIT)  explored  ADS  support  of  precision  guided  munitions  (PGM)  testing;  the 
End-To-End  (ETE)  Test  investigated  ADS  support  for  command,  control,  communications, 
computers,  intelligence,  surveillance,  and  reconnaissance  (C4ISR)  testing;  and  the  Electronic 


1  This  office  is  now  the  Deputy  Director,  Developmental  Test  and  Evaluation. 
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Warfare  (EW)  Test  examined  ADS  support  for  EW  testing.  JADS  also  observed,  or  participated 
at  a  modest  level  in,  ADS  activities  sponsored  and  conducted  by  other  agencies  in  an  effort  to 
broaden  conclusions  developed  in  the  three  dedicated  test  areas. 


2 


2.0  Supporting  Activities  and  Results 

2.1  Utility  for  Precision  Guided  Munitions  Testing 

The  SIT  investigated  the  ability  of  ADS  to  support  air-to-air  missile  testing.  The  test  included 
two  sequential  phases,  a  Linked  Simulators  Phase  (LSP)  and  a  Live  Fly  Phase  (LFP).  Both 
phases  incorporated  one-versus-one  scenarios  based  upon  profiles  flown  during  live  test  activities 
and  limited  target  countermeasure  capability. 

The  LSP  distributed  architecture  incorporated  four  nodes:  the  shooter,  an  F/A-18  manned 
avionics  laboratory  at  China  Lake,  California;  the  target,  an  F-14  manned  avionics  laboratory  at 
Point  Mugu,  California;  a  hardware-in-the-loop  (HWIL)  missile  laboratory  at  China  Lake  that 
hosted  an  air  intercept  missile  (AIM)-9M  missile;  and  a  test  control  center  initially  located  at 
Point  Mugu  and  later  relocated  to  the  JADS  facility  in  Albuquerque,  New  Mexico. 

The  LFP  distributed  architecture  linked  two  live  F-16  aircraft  (a  shooter  and  target)  on  the  Eglin 
Air  Force  Base,  Florida,  Gulf  Test  Range;  the  Eglin  Central  Control  Facility;  an  HWIL  missile 
laboratory  at  Eglin  that  hosted  an  AIM-120  missile;  and  a  test  monitoring  center  at  the  JADS 
facility  in  New  Mexico. 

2.1.1  System  Integration  Test  Results  and  Conclusions 

Within  the  narrow  confines  of  the  SIT  data,  our  assessment  is  that  the  two  architectures  we 
employed  have  utility  for  support  of  T&E.  The  JADS  data  indicate  that  activities  ranging  from 
parametric  analyses  to  integrated  weapons  system  testing  are  both  practical  and  cost  effective. 
Our  broad  conclusions  and  lessons  learned  can  be  summarized  as  follows. 

-  For  T&E  applications,  the  technology  is  not  at  the  “plug-and-play”  stage.  While  practical  and 
cost  effective  in  many  cases,  implementation  is  more  challenging  than  many  people  think. 
Plan  for  a  lot  of  rehearsals  and  “fix”  time. 

-  The  effects  of  latency  and  other  ADS-induced  errors  can  often  (not  always)  be  mitigated. 

-  Synchronization  is  as  much  a  challenge  as  latency. 

-  Instrumentation  and  data  management  are  challenges. 

-  ADS  has  great  potential  as  a  T&E  support  tool:  It  is  a  valuable  addition  to  the  tester’s  tool 
kit.  ADS  will  not  obviate,  but  in  some  cases  it  may  reduce,  the  need  for  live  testing. 

-  Our  data  suggest  test  savings  are  possible. 

2.1.2  Observations  for  Precision  Guided  Munitions  T&E 
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Through  a  process  of  inductive  reasoning  we  can  transfer  some  of  the  SIT-based  specifics  to  the 
general  class  of  PGM.  In  the  general  case,  the  elements  of  the  SIT  architectures  are  basic  to  all 
PGM  cases.  There  are  (1)  a  launch  platform  or  shooter,  (2)  a  PGM,  (3)  an  intended  target,  (4)  an 
operating  environment  (to  include  countermeasures),  and  (5)  a  test  control  center. 

The  shooter,  PGM,  and  target  can  be  represented  in  any  of  the  three  forms  associated  with 
distributed  simulation:  live,  virtual,  or  constructive.  SIT  looked  at  an  AIM-9  and  an  AIM-120. 
The  physical  dynamics  of  the  problem  are  comparable  with  any  class  of  PGM.  The  physics 
associated  with  detection,  tracking,  and  guidance  may  differ  significantly  depending  upon  bands, 
techniques,  and  the  operational  medium  a  missile  operates  in.  We  do  not  see  a  one-for-one 
transfer  of  SIT  techniques  to  other  tests.  Each  test  has  specific  requirements,  often  peculiar  to  the 
particular  SUT.  We  do  see  a  transfer  of  the  principles,  design  processes,  and  methodologies  used 
in  SIT. 

Countermeasures  were  only  represented  in  rudimentary  form  in  the  SIT,  but  we  see  no  technical 
impediments  at  the  conceptual  level  to  implementing  high-fidelity  countermeasures  in  ADS.  The 
devil  will  be  in  the  details,  and  costs  and  technical  challenges  will  be  very  case  specific.  Complex 
environmental  details  associated  with  atmospherics,  space,  oceanography,  etc.,  are  more 
challenging.  In  the  SIT  case,  the  LFP,  since  it  involved  flying  open  air,  incorporated  real 
atmospheric  effects. 

A  test  control  center  is  a  requirement  for  all  testing,  distributed  or  not.  Fortunately,  the  SIT 
experience  suggests  that  the  control  center  can  function  from  almost  anywhere.  The  inference  is 
that  an  existing  control  center  somewhere  may  well  meet  a  specific  tester’s  needs. 

The  SIT  program  was  budget  and  schedule  constrained.  Consequently,  there  were  important 
aspects  of  PGM  testing  which  SIT  did  not  explore.  From  a  single  shooter  perspective,  some  of 
these  included  multiple  launches  against  a  single  target,  single  launches  against  clustered  targets, 
and  multiple  launches  against  multiple  targets.  SIT  did  not  examine  few-on-few  or  many-on- 
many  scenarios.  Our  expectation,  unsupported  by  hard  data,  is  that  few-on-few  implementations 
are  possible.  The  difficulties  and  costs  would  be  extremely  sensitive  to  the  fidelity  requirements 
and  the  availability  of  existing  facilities,  e.g.,  HWIL  facilities  or  installed  systems  test  facilities. 

The  SIT  results  strongly  suggest  that  ADS  has  good  potential  for  improving  PGM  testing.  The 
implication  is  that  test  planners  should  consider  the  technology  as  a  relevant  tool  for  their 
program  until  an  objective  assessment  suggests  otherwise.  Bottom  line:  Know  ADS  is  there,  and 
assess  how,  or  if,  it  should  be  used  in  a  specific  program. 

2.2  Utility  for  Command,  Control,  Communications,  Computer,  Intelligence, 
Surveillance,  and  Reconnaissance  (C4ISR)  Testing 

The  ETE  Test  evaluated  the  utility  of  ADS  to  support  testing  of  C4ISR  systems.  The  test  used 
the  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one  component  of  a 
representative  C4ISR  system.  The  ETE  Test  also  evaluated  the  capability  of  the  JADS  Test 
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Control  and  Analysis  Center  (TCAC)  to  control  a  distributed  test  of  this  type  and  to  remotely 
monitor  and  analyze  test  results. 

The  ETE  Test  consisted  of  four  phases.  Phase  1  developed  or  modified  the  components  needed 
to  develop  the  ADS  test  environment.  Phase  2  used  the  ADS  test  environment  to  evaluate  the 
utility  of  ADS  to  support  developmental  test  and  evaluation  (DT&E)  and  early  operational  test 
and  evaluation  (OT&E)  of  a  C4ISR  system  in  a  laboratory  environment.  Phase  3  transitioned 
portions  of  the  architecture  to  the  E-8C  aircraft,  ensured  that  the  components  functioned 
properly,  and  checked  that  the  synthetic  environment  correctly  interacted  with  the  aircraft  and 
actual  light  ground  station  module  (LGSM).  Phase  4  evaluated  the  ability  to  perform  test  and 
evaluation  of  the  E-8C  and  LGSM  in  a  synthetically  enhanced  live  test  environment. 

The  test  concept  was  to  use  ADS  to  supplement  the  operational  environment  experienced  by  the 
E-8C  and  LGSM  operators.  By  mixing  available  live  targets  with  targets  generated  by  a 
constructive  model,  a  battle  array  approximating  the  major  systems  present  in  a  notional  corps 
area  of  interest  was  presented.  By  constructing  a  network  with  nodes  representing  appropriate 
command,  control,  communications,  computers  and  intelligence  (C4I)  and  weapon  systems,  a 
more  robust  cross  section  of  players  was  available  for  interaction  with  the  E-8C  and  LGSM 
operators. 

The  TCAC,  in  Albuquerque,  New  Mexico,  provided  test  control. 

The  ETE  Test  used  the  Janus  6.88D  simulation  to  generate  the  entities  representing  the  elements 
in  the  rear  of  a  threat  force.  The  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center 
(TRAC)  at  White  Sands  Missile  Range  (WSMR),  New  Mexico,  provided  the  Janus  scenario  feed. 

Bravo  Company,  303d  Military  Intelligence  Battalion  represented  the  LGSM  and  target  analysis 
cell  (TAC).  Fire  support,  in  the  form  of  the  Advanced  Field  Artillery  Tactical  Data  System 
(AFATDS),  was  represented  by  soldiers  from  the  4th  Infantry  Division  (Mechanized). 

Communications  among  these  systems  employed  such  doctrinally  correct  means  as  the  CGS-100, 
a  subsystem  of  the  Compartmented  All  Source  Analysis  System  (ASAS)  Message  Processing 
System  (CAMPS),  remote  workstations  (RWSs),  and  AFATDS  message  traffic. 

The  Tactical  Army  Fire  Support  Model  (TAFSM)  simulation  modeled  the  Army  Tactical  Missile 
System  (ATACMS)  battalion  and  sent  the  fire  and  detonate  protocol  data  units  (PDUs)  to  the 
Janus  6.88D  simulation.  Janus  then  modeled  the  engagement  results  and  reflected  them  in  the 
synthetic  environment. 

The  Joint  STARS  E-8C  simulation,  called  the  Virtual  Surveillance  Target  Attack  Radar  System 
(VSTARS),  represented  the  radar  subsystem  of  the  Joint  STARS  E-8C  in  a  laboratory 
environment.  It  was  composed  of  a  distributed  interactive  simulation  (DIS)  network  interface 
unit  (NIU),  a  radar  processor  simulator  and  integrator  (RPSI)  that  contained  the  two  real-time 
radar  simulations  with  necessary  databases,  and  various  simulations  of  E-8C  processes.  VSTARS 
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was  operated  at  the  Northrop  Grumman  Surveillance  and  Battle  Management  Systems  facility  in 
Melbourne,  Florida. 

2.2.1  End-to-End  Test  Results  and  Conclusions 

Within  the  narrow  confines  of  the  ETE  Test  data,  our  assessment  is  that  the  architecture  we 
employed  has  utility  for  support  of  C4ISR  T&E.  The  JADS  data  indicate  that  DT&E  and  OT&E 
activities  incorporating  ADS  technology  are  both  practical  and  cost  effective.  Our  broad 
conclusions  and  lessons  learned  can  be  summarized  as  follows. 

An  ADS  environment  can  enhance  C4ISR  system  DT&E  and  OT&E.  In  comparison  with 
conventional  tests,  ADS  allows  testers  to  examine  C4ISR  systems  under  realistic  conditions  for 
longer  periods  of  time,  over  far  larger  battlespaces,  and  at  a  much  lower  cost.  This  versatile 
technology  can  provide  test  environments  that  include  large  numbers  of  entities,  entities  operating 
under  realistic  but  unsafe  conditions,  and  joint  and  combined  operations.  ADS  provides  C4ISR 
system  testers  with  greater  flexibility  in  designing,  executing,  and  analyzing  their  tests.  During 
DT&E,  ADS  allows  for  more  realistic  compliance  testing  of  C4ISR  subsystems  and  efficient 
implementation  of  the  test-fix-verify  cycle  for  software  development. 

ADS  testing  provides  dramatic  abilities  to  test  C4ISR  systems  or  components  in  that  system.  For 
instance,  in  the  ETE  Test  ADS  environment,  a  developmental  test  could  be  performed  on  the 
Joint  STARS  operations  and  control  subsystem  using  VSTARS  as  a  stimulus.  With  a  similar 
configuration,  an  operational  test  could  be  accomplished  on  an  LGSM.  The  ETE  Test  ADS 
environment  also  provided  ample  opportunities  to  install  new  components  for  various  types  of 
testing.  Links  to  airborne  weapon  system  simulators,  complimentary  sensor  feeds  or  other 
command  and  control  structures  can  be  easily  accomplished.  The  development  of  an  ADS  test 
environment  during  system  development  greatly  improves  opportunities  for  C4ISR  system 
training  after  the  completion  of  the  test.  The  same  infrastructure  developed  for  testing  can  be 
modified  and  transitioned  to  a  training  environment  resulting  in  program  savings.  This  technology 
allows  C4ISR  system  operators  to  confirm  current  tactics,  try  “what-if  ’  scenarios  and  new  tactics, 
test  the  interoperability  and  compatibility  of  their  equipment,  and  gain  usefiil  experience  in  a 
realistic  operating  environment  containing  multiple  assets. 

The  ETE  Test  required  only  a  small  part  of  the  available  bandwidth  and  exhibited  a  low  PDU 
latency  rate.  The  ETE  Test  network  was  highly  reliable  during  testing,  due  largely  to  the  ETE 
Test  team’s  extensive  pretest  risk  reduction  efforts. 

Compared  to  conventional  testing,  ADS  testing  reduces  the  need  for  large  numbers  of  fielded 
personnel  and  vehicles.  The  ability  to  automatically  collect  and  analyze  test  data  also  reduces  the 
number  of  people  required  for  setup,  execution,  and  analysis.  ADS  test  success  relies  on  well- 
organized  test  control  and  data  management  procedures.  Distributed  testing  requires 
sophisticated  instrumentation,  trained  personnel  to  operate  and  maintain  that  equipment,  and 
funds  to  support  personnel  and  equipment  at  distant  test  nodes. 
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Testers  should  carefully  plan  the  development  of  the  simulations  and  links  comprising  their  ADS 
environment.  During  test  execution,  they  must  ensure  that  the  time  sources  are  synchronized  and 
continuously  monitor  PDU  traffic.  The  distributed  nature  of  ADS  testing  necessitates  special 
equipment  for  network  checkout  and  verification  and  requires  strict  configuration  control  of 
analysis  tools  and  collected  data. 

ADS  test  planning  should  be  detailed  enough  to  encompass  key  requirements  at  the  earliest 
possible  stages,  yet  flexible  enough  to  accommodate  unexpected  situations  during  test  execution. 

A  conservative  development  approach  is  recommended.  Accomplish  risk  reduction  activities 
before  each  ADS  test  and  let  each  ADS  test  build  on  the  success  of  earlier  experiments. 
Successful  test  execution  requires  effective  intemode  communication,  test  and  resource  control, 
and  data  management  procedures. 

2.2.2  Observations  for  C4ISR  T&E 

Through  a  process  of  inductive  reasoning,  we  can  transfer  some  of  the  specifics  of  the  ETE  Test 
to  the  general  class  of  C4ISR  systems.  In  the  general  case,  the  elements  of  the  ETE  Test 
architecture  are  basic  to  all  mission-level  testing  of  C4ISR  systems.  In  other  words,  there  are  a 
sensor,  a  shooter,  an  intended  target,  an  operating  environment,  and  a  test  control  center. 

The  sensor,  shooter,  and  target  can  be  represented  in  any  of  the  three  forms  associated  with 
distributed  simulation:  live,  virtual,  or  constructive.  We  do  not  see  a  one-for-one  transfer  of  ETE 
test  techniques  to  other  tests.  Each  test  has  specific  requirements,  often  peculiar  to  the  particular 
system  under  test.  However,  we  do  see  a  transfer  of  the  principles,  design  processes,  and 
methodologies  used  in  the  ETE  Test. 

Technically,  it’s  not  impossible  to  develop  a  high-fidelity  operating  environment  in  ADS. 
However,  the  costs  and  challenges  will  vary  from  test  to  test  and  will  depend  on  the  desired  level 
of  complexity. 

A  test  control  center  is  a  requirement  for  all  testing,  distributed  or  not.  Fortunately,  the  ETE  Test 
experience  suggests  that  the  control  center  can  function  from  almost  anywhere.  Thus,  a  specific 
test  may  save  resources  by  using  an  existing  control  center. 

The  ETE  Test  results  strongly  suggest  that  ADS  has  excellent  potential  for  improving  C4ISR 
testing.  Thus,  test  planners  should  consider  the  technology  as  a  relevant  tool  for  their  program, 
until  an  objective  assessment  suggests  otherwise. 

2.3  Utility  for  Electronic  Warfare  (EW)  Testing 

The  EW  Test  evaluated  the  utility  of  ADS  to  support  EW  T&E.  While  the  test  used  several 
efforts  to  examine  ADS-based  T&E,  the  cornerstone  effort  was  a  series  of  traditional  and  ADS- 
based  test  events  using  an  airborne  self-protection  jammer  (SPJ).  This  effort  was  called  the  SPJ 
test.  The  SPJ  test  defined  a  simple,  repeatable  test  scenario.  The  scenario  was  executed  in  three 
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traditional  test  environments  to  create  a  data  baseline.  The  test  scenario  was  then  executed  in  two 
ADS-enhanced  test  environments.  The  first  ADS-based  test  event  used  a  real-time  digital  system 
model  (DSM)  interacting  with  manned  threat  simulators  at  the  Air  Force  Electronic  Warfare 
Environment  Simulator  (AFEWES)  facility,  Fort  Worth,  Texas.  The  second  ADS-based  test  used 
the  SPJ  installed  on  an  F-16  suspended  in  the  anechoic  chamber  at  the  Navy’s  Air  Combat 
Environment  Test  and  Evaluation  Facility  (ACETEF),  Patuxent  River,  Maryland.  The  data  from 
all  tests  were  statistically  compared  in  an  attempt  to  quantify  the  impacts  of  ADS. 

The  other  efforts  used  by  JADS  to  examine  the  utility  of  ADS  are 

1)  The  Office  of  the  Secretary  of  Defense  (OSD)  CROSSBOW  Committee-sponsored  Threat 
Simulator  Linking  Activity  (TSLA)  effort, 

2)  The  Defense  Modeling  and  Simulation  Organization  (DMSO)-sponsored  High  Level 
Architecture  (HLA)  Engineering  Protofederation  (EPF)  effort,  and 

3)  The  Army’s  Advanced  Distributed  Electronic  Warfare  System  (ADEWS)  development  effort. 

Each  of  these  efforts  added  to  the  SPJ  test  experience  to  provide  JADS  with  a  broader 
understanding  of  the  utility  of  ADS  to  EW  T&E. 

2.3.1  EW  Test  Results  and  Conclusions 

Within  the  confines  of  the  SPJ  test  data,  JADS  concluded  that  ADS  architectures  that  allow  the 
capabilities  of  geographically  separated  facilities  to  be  combined  to  create  a  realistic  test 
environment  for  EW  devices  can  be  designed.  This  allows  the  same  test  environment  to  be  used 
for  SUT  representations  ranging  from  DSMs  to  operational  equipment.  Testing  in  a  common 
ADS-based  environment  represents  a  significant  departure  from  the  traditional  sequential  EW  test 
process. 

Key  Results 

-  Designing  ADS  architectures  requires  a  close  team  comprised  of  several  technical  experts 
spanning  several  disciplines  directed  by  a  system  integrator. 

-  The  architecture  produced  valid  results  for  both  the  DSM  and  actual  jammer  hardware. 

-  Latency  within  the  closed-loop  interaction  was  aggressively  managed,  and  JADS  was  able  to 
meet  its  objective  for  more  than  95%  of  the  runs. 

-  The  HLA  appears  to  be  a  feasible  method  for  linking  simulations  for  T&E.  It  is  appropriate  to 
use  HLA  to  link  to  other  HLA-compliant  simulations/simulators,  but  the  T&E  community 
should  not  view  it  as  the  only  architecture  to  consider  when  designing  distributed  tests. 

-  Two  of  the  eleven  EW  test  facilities  surveyed  in  1996  as  part  of  the  TSLA  effort  that  were 
appropriate  for  ADS-based  testing  have  been  closed.  While  this  is  a  significant  erosion  in  the 
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infrastructure  needed  to  design  and  execute  ADS-based  tests,  it  already  limits  the  traditional 
EW  testing  process. 

2.3.2  Observations  for  EW  T&E 

JADS  assessment,  based  on  the  different  EW  Test  efforts,  is  that  ADS  has  varying  levels  of  utility 
for  EW  T&E.  These  levels  of  utility  depend  on  the  nature  of  the  EW  device  being  tested  and  the 
availability  of  suitable  test  facilities.  Single  function  EW  devices  and  federated  EW  systems  are 
expected  to  benefit  least  from  an  ADS-enhanced  test  process.  Only  radio  frequency  (RF) 
jammers  may  see  sufficient  benefit  to  outweigh  the  additional  cost  of  an  ADS-enhanced  test 
process.  Integrated  EW  systems  may  see  significant  benefits  where  a  single  test  facility  is  not 
capable  of  providing  all  the  stimulation  (including  the  closed-loop  SUT  versus  manned  threat 
interaction  for  systems  that  include  RF  jammers)  needed  to  simultaneously  test  all  the  particular 
integrated  EW  system’s  functions.  Systems  of  systems  testing  such  as  that  required  for  electronic 
support  (ES)  systems  should  see  significant  benefits  in  ADS-based  testing. 

ADS  allows  the  test  designer  to  construct  tests  that  address  objectives  beyond  the  capability  of  a 
single  facility.  While  connecting  facilities  can  be  beneficial,  not  all  EW  testing  requires  the  use  of 
ADS.  Single  device  and  federated  systems  testing  are  generally  adequately  supported  by  the 
sequential  facility  testing  paradigm  (the  progression  from  DSM  to  system  integration  laboratory 
[SIL],  SIL  to  hardware-in-the-loop  [HITL],  HITL  to  installed  systems  test  facility  [ISTF],  ISTF 
to  open  air  range  [OAR])  recommended  in  Department  of  Defense  EW  test  process  description. 
Testing  integrated  EW  systems  and  systems  of  systems  does  not  readily  conform  to  the  sequential 
facility  testing  paradigm.  The  requirement  for  testing  these  systems  is  to  create  an  operationally 
realistic  single  environment  in  which  all  integrated  EW  functions,  including  closed-loop 
interaction  with  manned  threat  simulators,  are  simultaneously  tested  while  interacting  with  the 
host  platform,  the  host  platform  operator(s),  and  the  appropriate  blue  and  red  players.  Any  less 
stringent  test  increases  the  risk  of  fielding  a  flawed  system.  Highly  integrated  avionics  systems 
such  as  the  F-22  share  resources.  The  only  way  to  fully  understand  the  effects  of  resource  sharing 
is  to  test  all  system  functions  at  the  same  time  in  the  operationally  realistic  single  environment 
described  above. 

To  be  tested,  all  integrated  EW  systems  may  not  require  ADS.  If  a  single  facility  is  capable  of 
providing  all  the  stimulation  (including  the  closed-loop  SUT  versus  manned  threat  interaction  for 
systems  that  include  RF  jammers)  for  the  particular  integrated  EW  system,  then  ADS  is  not 
needed.  However,  if  no  facility  exists,  ADS  offers  an  alternative  to  potentially  expensive  facility 
modifications  or  expansion.  The  development  and  integration  of  distributed  test  facilities  (such  as 
AFEWES  and  ACETEF)  capable  of  simultaneous  EW  sensors  stimulation  and  closed-loop  threat 
interaction  present  the  facility  designer  with  the  same  problems  of  latency,  data  loss,  and  out-of- 
order  data  that  confront  the  ADS  architecture  designer. 

Systems  of  systems  testing  is  not  likely  to  be  practical  or  affordable  in  a  single  facility  or  on  a 
single  OAR.  Simple  cooperative  jamming  tests  against  single  threats  can  be  accomplished  using 
the  sequential  testing  methods  described  in  the  OSD  EW  test  process.  More  complicated  systems 


9 


of  systems  testing  on  an  OAR  is  constrained  by  the  limitations  the  single  facility  or  the  limitations 
of  the  range  selected  and  the  cost  of  bringing  the  required  platforms  together  for  the  test  event. 

The  EW  Test  results  stongly  suggest  that  ADS  has  good  potential  for  improving  some  types  of 
EW  testing.  The  implication  is  that  test  planners  should  consider  the  technology  as  a  relevant  tool 
for  their  program  until  an  objective  assessment  suggests  otherwise. 


10 


3.0  Overall  Utility 

3.1  General  Role  of  Distributed  Testing 

Roles  for  distributed  testing  can  be  identified  for  both  DT&E  (in  which  the  general  objective  is  to 
determine  the  SUT’s  ability  to  meet  specifications  using  preproduction  subsystems)  and  OT&E 
(in  which  the  general  objective  is  to  determine  the  SUT’s  ability  to  meet  user’s  operational 
effectiveness  and  suitability  requirements  using  production-level  systems  in  realistic  combat 
scenarios).  The  underlying  ADS  technology  can  support  DT&E  and  OT&E  equally  well. 

3.1.1  DT&E 

Early  DT&E  of  complex  systems  frequently  focuses  on  stand-alone  components,  subsystems,  or 
systems  testing.  This  testing  is  essential  and  in  most  cases  can  be  adequately  conducted  in  stand¬ 
alone  test  facilities.  Distributed  testing  becomes  a  valuable  DT&E  tool  when  you  start  to 
integrate  the  components,  subsystems  and  systems  together.  The  use  of  ADS  can  result  in  much 
better  system  integration  evaluations.  High-fidelity  integration  can  begin  prior  to  having 
flight/soldier-ready  systems.  In  those  cases  where  live  only  testing  is  deemed  necessary  in 
addition  to  distributed  testing,  the  distributed  test  environment  can  be  used  to  provide  realistic 
test  rehearsals,  to  refine  test  scenarios,  and  to  verify  data  collection  and  analysis  equipment  and 
processes.  Distributed  testing  can  also  ease  the  transition  to  OT&E  testing.  OT&E  test  scenarios 
are  frequently  more  complex  than  DT&E  scenarios.  Distributed  testing  provides  the  capability  to 
affordably  test  early  versions  of  the  system  under  test  in  a  complex  synthetic  test  environment. 
This  capability  not  only  eases  the  transition  to  OT&E,  but  more  importantly,  it  identifies  failure 
modes  earlier  in  the  test  process  when  they  can  be  fixed  cheaper  and  in  less  time  than  when  they 
are  discovered  in  OT&E. 

3.1.2  OT&E 

The  key  contribution  of  distributed  testing  to  OT&E  is  that  it  enables  affordable  system  of 
systems  or  mission-level  testing.  Currently,  cost  and  safety  constraints  limit  the  density  of  forces 
in  typical  OT&E  to  much  less  than  would  be  found  in  battle.  Also,  it  is  usually  impossible  to 
obtain  the  actual  threat  systems  that  the  system  under  test  would  operate  against  and  have  them 
operate  in  a  realistic  “unconstrained”  manner.  The  use  of  distributed  testing  can  greatly  augment 
the  force  density  during  testing  with  either  manned  simulators  or  DSMs  representing  the 
additional  forces.  When  used  in  conjunction  with  live  only  tests,  distributed  testing  can  be  used  to 
provide  realistic  test  rehearsal,  including  refinement  of  live  fire  scenarios  in  order  to  ensure  test 
objectives  are  met.  It  can  also  be  used  for  post-test  analysis  and  can  aid  in  resolving  anomalies 
observed  in  live  fire  testing. 

3.1.3  Acquisition  Process 

In  the  not  too  distant  past,  the  acquisition  process  was  marked  by  waterfall  development  of 
systems  and  required  independent  DT&E  and  OT&E.  Clearly  the  trend  in  acquisition  is  to  spiral 
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development,  increase  contractor  testing,  and  combine  or  merge  DT&E  and  OT&E.  Simulation 
Based  Acquisition  (SBA)  is  designed  to  accelerate  this  trend.  Distributed  testing  is  ideally  suited 
to  support  these  changes  in  the  acquisition  process  because  it  allows  evolving  SUT 
representations  to  be  tested  in  a  common  synthetic  test  environment  throughout  the  entire 
acquisition  process.  The  system  under  test  and  the  test  assets  (live,  virtual,  and/or  constructive) 
that  make  up  the  synthetic  test  environment  can  be  distributed  over  large  geographic  distances.  If 
the  system  under  test  is  itself  a  system  of  systems,  these  systems  can  also  be  geographically 
distributed  and  linked  to  the  synthetic  test  environment.  This  supports  a  continuum  of 
development  and  testing  as  opposed  to  discrete  test  events  and  milestones.  It  allows  specification 
testing  and  requirements  testing  to  be  conducted  in  the  same  test  environment  that  will  allow  the 
relationship  between  specifications  and  requirements  to  be  closely  defined.  The  linked 
environment  also  will  support  collaborative  engineering  and  collaborative  analysis  within  and 
between  contractor  and  government  organizations  independent  of  location.  Maximum  benefits 
and  cost  savings  will  only  be  realized  when  distributed  testing  is  used  throughout  the  entire 
acquisition  process  from  requirements  definition  to  training. 

3.2  General  Utility  Assessment 

3.2.1  Benefits 

Three  areas  or  categories  of  benefits  were  identified  and  demonstrated  by  the  JADS  JT&E. 

The  ability  to  overcome  current  test  limitations.  During  the  feasibility  study  prior  to  the  JADS 
JT&E,  a  survey  of  existing  conventional  development  and  operational  test  limitations  was 
conducted.  Three  hundred  and  sixty-one  limitations  were  extracted  from  test  reports  and  test  and 
evaluation  master  plans.  A  triservice  panel  of  testers  was  convened  to  validate  the  limitations  and 
loosely  rank  them.  The  top  developmental  and  operational  limitations  are  presented  below.  A 
double  check  signifies  that  JADS  demonstrated  the  capability  to  overcome  that  limitation  in  one 
or  more  of  the  JADS  test  cases.  A  single  check  signifies  an  inferred  capability  to  overcome  the 
test  limitation. 

Potential  for  ADS  to  overcome  traditional  developmental  test  (DT)  limitations 

VV  Inability  to  integrate  avionics  testing 

V  Limitations  to  testing  special  access  programs 

VV  Incompatibility  of  collected  data 

VV  Human  interaction  not  represented 

VV  Nonrepresentative  force  levels 

VV  Inadequate  quantity  and  types  of  threat  systems 
VV  Electronic  combat  testing  not  allowed,  is  limited  or  restricted 

V  Lack  of  systems  for  compatibility  testing 
W  Real-time  modeling/simulation  not  available 
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Potential  for  ADS  to  overcome  traditional  operational  test  (OT)  limitations 

VV  Inadequate  quantity  and  types  of  targets 

VV  Inadequate  quantity  and  types  of  threat  systems 

VV  Inadequate  quantity  and  types  of  friendly  systems 

V  Electronic  combat  testing  not  allowed,  is  limited  or  restricted 

VV  Nonrepresentative  force  levels 

VV  Human  interaction  not  represented 

VV  Insufficient  test  articles 

VV  Unrealistic  test  scenarios 

VV  Insufficient  number  of  test  events 

Clearly  ADS  has  the  potential  to  overcome  many  conventional  test  limitations. 

The  ability  to  identify  failure  modes  earlier.  ADS  provides  the  capability  to  link  a  representation 
of  the  SUT  to  a  distributed  test  environment.  The  distributed  test  environment  can  be  held 
relatively  stable  while  the  SUT  evolves  and  matures  through  its  development  cycle.  The  first 
representation  of  the  SUT  may  (should)  be  a  digital  system  model,  followed  by  prototype 
hardware  and  software  in  the  contractor’s  laboratory,  followed  by  a  full-up  system  in  an  installed 
systems  test  facility.  The  ability  to  test  evolving  SUT  representations  in  a  common  test 
environment  has  several  advantages.  First,  it  reduces  the  number  of  changing  variables  across  test 
phases.  In  conventional  testing,  it’s  not  unusual  to  change  the  test  environment  when  the  SUT 
representation  changes.  Another  significant  advantage  is  the  ability  to  identify  SUT  failure  modes 
earlier  in  the  development  cycle  when  they  can  be  fixed  quicker  and  more  cheaply.  ADS  is 
particularly  well  suited  to  identifying  interoperability  problems  early  in  the  acquisition  process, 
especially  when  real-world  communications  links  are  an  integral  part  of  the  linked  test 
configuration.  The  ability  to  create  a  robust  test  environment  with  realistic  sensor  and 
communications  inputs  also  enables  the  early  identification  of  hardware  and/or  software  data 
processing  problems.  The  ability  to  incorporate  human  operators  early  allows  human  factors 
issues,  concepts  of  operations,  tactics,  techniques  and  procedures,  and  training  to  all  be  resolved 
before  OAR  testing  begins.  The  ability  to  identify  and  address  suitability  issues  early  is  limited  by 
the  realism  of  laboratory  and  installed  systems  test  facility  operating  environments.  Linking  to 
SUTs  in  climatic  laboratories  is  being  investigated  and  will  improve  this  capability. 

The  ability  to  conduct  end-to-end  testing.  The  accomplishment  of  most  military  tasks  requires  the 
successful  coordination  and  integration  of  several  systems.  Decision  makers  are  asking  for 
mission-level  or  system  of  systems  test  information  in  order  to  support  acquisition  milestones. 
Conventional  testing  has  focused  on  individual  system  tests  and  makes  significant  assumptions 
concerning  the  performance  and  interoperability  of  other  systems.  ADS  provides  the  capability  to 
link  and  test  the  capability  of  a  system  of  systems  to  successfully  execute  a  mission  task  within  a 
single  test  environment.  This  approach  has  fewer  assumptions  and  requires  fewer  leaps  of  faith  in 
the  analysis  and  evaluation  of  test  results.  The  JADS  ETE  Test  is  an  excellent  example. 
Historically  the  Joint  STARS  (E-8C  aircraft  and  common  ground  station),  command  and  control 
nodes  (TAC  and  fire  support  element)  and  the  ATACMS  weapon  systems  would  all  be  tested 
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separately.  Analysis  would  be  used  to  combine  the  results  and  predict  the  number  of  target  kills 
supported  by  Joint  STARS.  The  JADS  ETE  Test  linked  these  systems  together  and  created  a 
single  test  environment  that  provided  much  greater  insight  into  interoperability,  timelines,  tactics, 
techniques  and  procedures  (TTPs),  and  traceability  of  causes  and  effects. 

3.2.2  Costs 

Costs  can  be  broken  into  two  categories  -  cost  savings  and  cost  avoidance.  The  MITRE 
Corporation  developed  an  ADS  in  T&E  work  breakdown  structure  (WBS)  in  support  of  JADS. 
The  top-level  WBS  is  presented  in  Table  1.  The  ADS  cost  impact  column  is  an  assessment  of 
costs  relative  to  a  conventional  test.  The  increased  complexity  of  an  ADS  test  results  in  a  small  to 
medium  increase  in  the  cost  of  most  categories.  The  cost  drivers  under  the  design  and 
development  category  are  simulations  and  interfaces.  If  a  new  or  modified  simulation  must  be 
developed,  the  cost  can  be  medium  to  high.  A  significant  cost  to  the  JADS  ETE  Test  was  the 
development  of  a  high-fidelity  representation  of  the  Joint  STARS  radar  system.  Both  Simulation 
Based  Acquisition  and  the  simulation,  test,  and  evaluation  process  advocate  that  the  system 
developer  be  required  to  deliver  a  digital  system  model  early  in  the  acquisition  process  and 
maintain  and  update  the  model  throughout  the  system  life  cycle.  This  should  reduce  or  eliminate 
the  cost  of  this  simulation  to  testers.  The  cost  impact  of  a  new  or  modified  interface  based  on 
JADS  experience  is  judged  to  be  medium  to  medium  high.  It  should  be  noted  that 
communications  links  were  not  found  to  be  significant  cost  drivers,  and  that  these  costs  are 
expected  to  decrease  in  the  future.  The  cost  impact  on  test  execution  and  analysis  runs  the  gamut 
from  medium  increases  to  large  cost  savings.  The  cost  of  test  assets  is  directly  related  to  the 
fidelity  of  the  asset  representation.  In  most  cases  a  live  asset  costs  the  most,  a  virtual  asset  less, 
and  a  constructive  asset  the  least.  ADS  allows  the  tester  to  obtain  the  required  fidelity  at 
minimum  cost.  A  related  issue  that  can  result  in  cost  savings  is  test  asset  availability.  In  this  case, 
live  test  assets  are  generally  the  least  readily  available,  virtual  more,  and  constructive  the  most 
available.  Availability  of  test  assets  drives  the  test  schedule,  especially  when  modifications  and 
retesting  are  required. 


Table  1.  ADS  Cost  Impact 


WBS  Element 

ADS  Cost  Impact 

Planning 

Low  increase 

Concept  Development 

Low  -  medium 
increase 

Design  and  Development 

Medium  increase 

Installation,  Integration  and  Test 

Medium  increase 

Text  Execution  and  Analysis 

Large  decrease  - 
low  increase 

Cost  avoidance  can  be  thought  of  as  unplanned  for  cost  savings.  ADS  supports  or  enhances  cost 
avoidance  in  several  ways.  By  identifying  SUT  problems  earlier,  you  can  not  only  avoid  the 
expense  associated  with  fixing  the  problems  later,  you  also  avoid  the  expense  of  failing  a  more 
expensive  test  later.  Similarly,  the  ability  to  identify  TTPs,  training,  data  processing,  and  analysis 
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problems  earlier  can  help  avoid  the  cost  of  expensive  OAR  test  failures  later.  The  Live  Fly  Phase 
of  the  SIT  provided  a  good  example  of  cost  avoidance  that  could  be  attributed  to  ADS.  The 
advanced  medium  range  air-to-air  missile  (AMRAAM)  initial  operational  test  and  evaluation 
failed  a  simultaneous,  multiple  shot  test  event.  The  cause  was  an  interoperability  problem 
concerning  the  rear  data  link  information  going  from  the  launch  aircraft  fire  control  radar  to  the 
AMRAAM.  This  interoperability  problem  would  have  been  found  in  a  LFP  ADS  test.  The  costs 
that  would  have  been  avoided  include  numerous  tests  and  missiles  to  demonstrate  that  the  fix 
worked;  the  dollars  associated  with  the  program  delay;  and  the  additional  gray  hairs  for  the 
program  manager  and  test  manager.  Cost  savings  will  be  very  program  specific  and  closely 
related  to  the  complexity  of  the  test  environment  (potential  for  increased  savings)  and  the  number 
of  new  simulations  and  interfaces  that  need  to  be  developed  instead  of  being  reused.  The 
potential  for  cost  avoidance  can  be  judged  to  be  high,  but  the  ability  to  predict  the  amount  for  any 
given  program  is  low. 

3.3  General  Concerns,  Constraints  and  Limitations 

3.3.1  Concerns 

When  JADS  entered  the  joint  test  there  were  several  areas  of  concern.  The  technical  concerns 
were  network  latency,  network  data  quality,  and  network  reliability.  Programmatic  concerns 
were  network  costs  and  verification,  validation  and  accreditation  (W &A)  costs.  A  key  finding  of 
the  JADS  program  is  that  in  most  cases  these  areas  were  indeed  only  concerns.  They  never 
became  constraints  or  limitations  in  the  execution  of  our  tests  and  shouldn’t  be  for  future 
programs  if  the  JADS  test  planning  methodologies  are  followed.  Costs  remain  a  concern.  Most 
existing  facilities  were  not  designed  to  be  linked  and  require  some  modification  along  with 
appropriate  interfaces  before  linking  is  possible.  This  cost  can  be  significant  but  can  be  amortized 
over  all  phases  of  the  acquisition  process. 

3.3.2  Constraints  and  Limitations 

The  lower  limit  of  latency  is  fixed  by  the  speed  of  light.  A  small  percentage  of  test  programs 
involve  very  tightly  coupled  interactions  between  the  SUT  and  other  players  in  which  latency  will 
be  a  constraint.  In  some  cases  these  limitations  can  be  overcome  by  collocating  critical 
components  of  the  interacting  systems,  as  done  in  the  EW  Test,  but  this  will  not  always  be 
possible.  Other  technical  constraints  are  the  ability  to  present  representations  of  synthetic  targets 
to  live  players  and  a  limited  ability  to  represent  dynamic  environmental  effects  among  live  and 
virtual  players.  These  can  be  considered  exotic  constraints  and  limitations  because  they  will  not 
occur  for  most  programs.  Currently,  the  most  prevalent  distributed  testing  constraint  is  the 
availability  and  capability  of  current  simulations.  An  assumption  at  the  beginning  of  the  joint  test 
was  that  high-fidelity  constructive,  virtual  and  live  simulations  existed  and  that  the  challenge 
would  be  in  linking  them  together.  The  JADS  experience  has  been  that  this  was  not  an  accurate 
assumption.  There  is  not  an  overabundance  of  live,  virtual,  or  high-fidelity  constructive 
simulations  and  those  that  do  exist  are  heavily  used.  The  challenge  for  JADS  was  as  much  to  find 
and  schedule  these  test  assets  as  to  link  them  together.  This  constraint  was  exacerbated  by  the 
lack  of  common  business  practices  across  services  and  between  test  facilities  and  test  ranges. 
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While  not  a  show  stopper,  this  lack  of  common  business  practices  hinders  the  efficiency  with 
which  distributed  testing  can  be  conducted,  adding  unnecessary  time  and  cost  overhead  burdens 
to  the  process. 

3.4  General  Requirements 

3.4.1  Standards 

DMSO,  the  Simulation  Interoperability  Standards  Organization  (SISO),  Foundation  Initiative 
2010,  and  many  other  government  and  commercial  organizations  are  developing  open  systems 
standards  for  technical  interoperability  among  live,  virtual,  and  constructive  simulations. 
Operational  system  interoperability  standards  are  also  being  developed,  and  the  distinction 
between  operational  interoperability  standards  and  simulation  interoperability  standards  is  very 
slight  in  many  cases.  Multilevel  security  standards  and  distributed  test  control  standards  are  areas 
which  require  additional  attention.  Although  technical  standards  have  a  long  way  to  go,  an 
adequate  process  appears  to  be  in  place  to  develop  them.  A  similar  process  needs  to  be  put  in 
place  to  develop  programmatic  standards  that  will  support  distributed  testing  across  all  phases  of 
the  acquisition  process.  As  in  the  technical  standards,  these  programmatic  standards  need  to  also 
support  international  distributed  testing  and  training. 

3.4.2  Networking 

Network  requirements  (i.e.,  expected  data  rate,  latency  budget,  communications  protocols, 
control  and  management  of  the  network)  need  to  be  defined  early  in  the  process.  These 
requirements  must  be  clearly  defined  and  forwarded  to  the  Defense  Information  Systems  Agency 
(DISA)  for  evaluation  of  how  they  can  best  support  the  requirements,  either  through  DISA 
common  user  networks  (i.e.,  Defense  Simulation  Internet  (DSI),  Secret  Internet  Protocol  Router 
Network  (SIPRNET),  Defense  Research  and  Engineering  Network  (DREN),  etc.),  or  by  granting 
a  waiver  exempting  use  of  common  user  networks  in  order  to  build  a  private  network  suitable  for 
the  requirements.  It  should  be  noted  that  if  DISA’ s  common  user  networks  are  used,  DISA  will 
allow  connection  to  only  one  of  the  networks.  These  networks  cannot  be  interconnected  because 
of  security,  interoperability,  and  tariff  constraints.  Thus,  it  requires  close  coordination  with  DISA 
to  ensure  the  network  of  choice  will  support  all  requirements. 

Time  is  of  the  essence.  Careful  planning  and  consideration  must  be  given  to  the  schedule  for 
implementing  the  communications  network.  On  average,  once  the  requirements  are  defined  and  a 
networking  solution  is  decided  upon,  it  will  take  a  minimum  of  120  days  (if  a  DISA  common  user 
network  is  to  be  utilized,  it  could  take  more  than  180  days)  to  procure  and  install  the  necessary 
communications  circuits  and  networking  hardware.  Also,  it  will  take  a  minimum  of  90  days  to 
obtain  the  necessary  communications  security  (COMSEC)  equipment  and  keying  material  to 
encrypt  the  data.  It  may  take  longer  if  a  COMSEC  subaccount  needs  to  be  established.  In 
addition,  time  must  be  allocated  in  the  schedule  to  install,  test,  and  validate  communications 
network  performance.  On  average,  JADS  allocated  one  week  per  site  to  accomplish  these  tasks. 
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3.4.3  Expertise 


Distributing  testing  requires  expertise  in  many  disciplines,  and  the  JADS  experience  has  been  that 
very  few  organizations  have  the  required  depth  and  breadth  of  expertise  required  to  efficiently 
conduct  a  successful  distributed  test.  JADS  had  to  develop  a  combination  of  government  and 
contractor  expertise  to  support  each  of  the  JADS  tests.  In  doing  so,  centers  of  distributed  testing 
expertise  were  established,  but  both  the  government  and  contractor  expertise  is  very  perishable. 
There  is  a  critical  requirement  to  establish  persistent  centers  of  excellence  within  OSD  and  the 
services  to  support  distributed  testing,  or  we  will  be  required  to  reinvent  the  proverbial  wheel 
many  more  times. 

3.4.4  Leadership  Support 

Continued  leadership  support  is  required  to  facilitate  the  use  of  distributed  testing  as  a  widely 
accepted  and  used  test  tool.  In  particular,  middle  management  needs  to  be  strongly  encouraged 
to  seriously  consider  the  use  of  distributed  testing  to  support  future  complex  acquisition 
programs.  The  previously  listed  requirements  are  areas  that  require  immediate  and  sustained 
leadership  support. 
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4.0  Summary 


The  utility  of  distributed  testing  is  case  specific,  but  for  many  complex  systems,  we  believe  it 
provides  the  only  affordable  means  of  conducting  adequate  system  of  systems  or  mission-level 
testing.  We  see  a  big  payoff  for  tests  that  involve  systems  integration,  interoperability,  and 
jointness.  Distributed  testing  also  appears  to  be  essential  to  support  current  trends  in  acquisition 
reform.  Current  technology,  in  most  cases,  is  adequate  to  support  distributed  testing  and  the 
technology  is  evolving  rapidly.  Current  business  practices,  however,  are  not  adequate  and  need  to 
be  improved  to  gain  maximum  utility  from  this  powerful  new  tool. 
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5.0  Abbreviations  and  Acronym  List 

AIM  air  intercept  missile 

ACETEF  Air  Combat  Environment  Test  and  Evaluation  Facility,  Patuxent  River, 
Maryland;  Navy  facility 

ADEWS  Advanced  Distributed  Electronic  Warfare  System;  Army  sponsored 

ADS  advanced  distributed  simulation 

AFATDS  Advanced  Field  Artillery  Tactical  Data  System 

AFEWES  Air  Force  Electronic  Warfare  Evaluation  Simulator,  Fort  Worth,  Texas;  Air 
Force  managed  with  Lockheed  Martin  Corporation 
AMRAAM  advanced  medium  range  air-to-air-missile 
ASAS  All  Source  Analysis  System 

ATACMS  Army  Tactical  Missile  System 

CAMPS  Compartmented  ASAS  Message  Processing  System 

C4I  command,  control,  communications,  computers  and  intelligence 

C4ISR  command,  control,  communications,  computers,  intelligence,  surveillance,  and 

reconnaissance 

COMSEC  communications  security 

CROSSBOW  Office  of  the  Secretary  of  Defense  committee  under  the  Director,  Test,  Systems 
Engineering  and  Evaluation 
DIS  distributed  interactive  simulation 

DISA  Defense  Information  Systems  Agency 

DMSO  Defense  Modeling  and  Simulation  Organization,  Alexandria,  Virginia 

DREN  Defense  Research  and  Engineering  Network 

DSI  Defense  Simulation  Internet 

DSM  digital  system  model 

DT  developmental  test 

DT&E  developmental  test  and  evaluation 

EPF  engineering  protofederations 

ES  electronic  support 

ETE  JADS  End-To-End  Test 

EW  electronic  warfare;  JADS  Electronic  Warfare  Test 

HI, A  high  level  architecture 

HITL  hardware-in-the-loop 

HWIL  hardware-in-the-loop 

ISTF  installed  systems  test  facility 

JADS  Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

Janus  interactive,  computer-based  simulation  of  combat  operations 

Joint  STARS  Joint  Surveillance  Target  Attack  Radar  System 

JT&E  joint  test  and  evaluation 

LGSM  light  ground  station  module 

LFP  live  fly  phase 

LSP  linked  simulators  phase 

M&S  modeling  and  simulation 
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MITRE 

company  that  provides  engineering  services 

MODSAF 

modular  semi-automated  forces 

NIU 

network  interface  unit 

OAR 

open  air  range 

OSD 

Office  of  the  Secretary  of  Defense 

OT 

operational  test 

OT&E 

operational  test  and  evaluation 

PDU 

protocol  data  unit 

PGM 

precision  guided  munitions 

RF 

radio  frequency 

RPSI 

radar  processor  simulator  and  integrator 

RWS 

remote  workstations 

SBA 

Simulation  Based  Acquisition 

SIPRNET 

Secret  Internet  Protocol  Router  Network 

SISO 

Simulation  Interoperability  Standards  Organization 

SIL 

system  integration  laboratory 

SIT 

system  integration  test;  JADS  System  Integration  Test 

SPJ 

self-protection  jammer 

SUT 

system  under  test 

T&E 

test  and  evaluation 

TAC 

target  analysis  cell 

TAFSM 

Tactical  Army  Fire  Support  Model 

TCAC 

Test  Control  and  Analysis  Center,  Albuquerque,  New  Mexico 

TRAC 

Army  Training  and  Doctrine  Command  Analysis  Center 

TSLA 

Threat  Simulator  Linking  Activity 

TTPs 

tactics,  techniques,  and  procedures 

VSTARS 

Virtual  Surveillance  Target  Attack  Radar  System 

W&A 

verification,  validation,  and  accreditation 

WBS 

work  breakdown  structure 

WSMR 

White  Sands  Missile  Range,  New  Mexico 
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